Enhancing 3D human pose estimation with NIR single-pixel imaging and time-of-flight technology: a deep learning approach.

The extraction of 3D human pose and body shape details from a single monocular image is a significant challenge in computer vision. Traditional methods use RGB images, but these are constrained by varying lighting and occlusions. However, cutting-edge developments in imaging technologies have introduced new techniques such as single-pixel imaging (SPI) that can surmount these hurdles. In the near-infrared spectrum, SPI demonstrates impressive capabilities in capturing a 3D human pose. This wavelength can penetrate clothing and is less influenced by lighting variations than visible light, thus providing a reliable means to accurately capture body shape and pose data, even in difficult settings. In this work, we explore the use of an SPI camera operating in the NIR with time-of-flight (TOF) at bands 850-1550 nm as a solution to detect humans in nighttime environments. The proposed system uses the vision transformers (ViT) model to detect and extract the characteristic features of humans for integration over a 3D body model SMPL-X through 3D body shape regression using deep learning. To evaluate the efficacy of NIR-SPI 3D image reconstruction, we constructed a laboratory scenario that simulates nighttime conditions, enabling us to test the feasibility of employing NIR-SPI as a vision sensor in outdoor environments. By assessing the results obtained from this setup, we aim to demonstrate the potential of NIR-SPI as an effective tool to detect humans in nighttime scenarios and capture their accurate 3D body pose and shape.

Deep-learning blurring correction of images obtained from NIR single-pixel imaging.

In challenging scenarios characterized by low-photon conditions or the presence of scattering effects caused by rain, fog, or smoke, conventional silicon-based cameras face limitations in capturing visible images. This often leads to reduced visibility and image contrast. However, using near-infrared (NIR) light within the range of 850-1550 nm offers the advantage of reduced scattering by microparticles, making it an attractive option for imaging in such conditions. Despite NIR's advantages, NIR cameras can be prohibitively expensive. To address this issue, we propose a vision system that leverages NIR active illumination single-pixel imaging (SPI) operating at 1550 nm combined with time of flight operating at 850 nm for 2D image reconstruction, specifically targeting rainy conditions. We incorporate diffusion models into the proposed system to enhance the quality of NIR-SPI images. By simulating various conditions of background illumination and droplet size in an outdoor laboratory scenario, we assess the feasibility of utilizing NIR-SPI as a vision sensor in challenging outdoor environments.

Single-pixel imaging: An overview of different methods to be used for 3D space reconstruction in harsh environments.

Different imaging solutions have been proposed over the last few decades, aimed at three-dimensional (3D) space reconstruction and obstacle detection, either based on stereo-vision principles using active pixel sensors operating in the visible part of the spectra or based on active Near Infra-Red (NIR) illumination applying the time-of-flight principle, to mention just a few. If extremely low quantum efficiencies for NIR active illumination yielded by silicon-based detector solutions are considered together with the huge photon noise levels produced by the background illumination accompanied by Rayleigh scattering effects taking place in outdoor applications, the operating limitations of these systems under harsh weather conditions, especially if relatively low-power active illumination is used, are evident. If longer wavelengths for active illumination are applied to overcome these issues, indium gallium arsenide (InGaAs)-based photodetectors become the technology of choice, and for low-cost solutions, using a single InGaAs photodetector or an InGaAs line-sensor becomes a promising choice. In this case, the principles of Single-Pixel Imaging (SPI) and compressive sensing acquire a paramount importance. Thus, in this paper, we review and compare the different SPI developments reported. We cover a variety of SPI system architectures, modulation methods, pattern generation and reconstruction algorithms, embedded system approaches, and 2D/3D image reconstruction methods. In addition, we introduce a Near Infra-Red Single-Pixel Imaging (NIR-SPI) sensor aimed at detecting static and dynamic objects under outdoor conditions for unmanned aerial vehicle applications.